word embedding and pre-training
Word Embeddings and Pre-training for Large Language Models (BERT, GPT)
In 2013, a paper was publish by Mikolov et al., that defined a Word2Vec model; its goal was to define words into a vectorized representation in some vector space based on its pretraining data (there are generally 2 variants CBOW and skip gram for Word2Vec). This was great for Deep learning models because you would take the text data and feed it into Word2Vec model (pre-trained on some large dataset), the result is a vector representation of the word. Then, use these vectors and pass into a DNN model (of some variant, say RNN/LSTM); the result is that we rely on the model to capture context representations in our data to acomplish a downstream task, say classification. Full responsibility is given to the model to learn context but not word representations, that was handled by Word2Vec. The key point to understand here is that words that have same spellings can mean differently upon the context, these words are called Homographs.